Encoded Executable File Detection Technique via Executable File Header Analysis*

Recently, the attack trends have been changed from fast and widespread malware propagation attacks to more sophisticated “targeted” attacks such as spy/adware, password stealers, ransom-ware, and botenets etc. and the attacks are tried via the automated malwares. In this situation, the malware is the most powerful weapon for the attackers. So, the attackers do not want their malwares to be reviled by anti-virus analyzer. In order to conceal their malware, malware programmers are getting utilize the anti reverse engineering techniques and code changing techniques such as the packing, encoding and encryption techniques. If the malware is packed or encrypted, then it is very difficult to analyze. Therefore, to prevent the harmful effects of malware and to generate signatures for malware detection, the packed and encrypted executable codes must initially be unpacked. The first step of unpacking is to detect the packed executable files. In this paper, a packed file detection technique based on a PE Header Analysis is proposed. In many cases, to pack and unpack the executable codes, PE files have unusual attributes in their PE headers. In this paper, these characteristics are utilized to detect the packed files. A Characteristic Vector (CV) that consists of eight elements is defined, and the Euclidean distance (ED) of the CV is calculated. The EDs of the packed files are calculated and represent the base threshold for the detection of packed files.